Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP). In this study, we re-visit the widely accepted notion in NLP that continued pre-training LMs on task-related texts improves the performance of fine-tuning (FT) in downstream tasks. Through experiments on eight single-sentence tasks and eight sentence-pair tasks in both semi-supervised and fully-supervised settings, we find that conventional continued pre-training does not consistently provide benefits and can even be detrimental for sentence-pair tasks or when prompt-based FT is used. To tackle these issues, we propose Prompt-based Continued Pre-training (PCP), which combines the idea of instruction tuning with conventional continued pre-training. Our approach aims to improve the performance of prompt-based FT by presenting both task-related texts and prompt templates to LMs through unsupervised pre-training objectives before fine-tuning for the target task. Our empirical evaluations on 21 benchmarks demonstrate that the PCP consistently improves the performance of state-of-the-art prompt-based FT approaches (up to 20.1% absolute) in both semi-supervised and fully-supervised settings, even with only hundreds of unlabelled examples. Additionally, prompt-based FT with PCP outperforms state-of-the-art semi-supervised approaches with greater simplicity, eliminating the need for an iterative process and extra data augmentation. Our further analysis explores the performance lower bound of the PCP and reveals that the advantages of PCP persist across different sizes of models and datasets.

Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP). In this study, we re-visit the widely accepted notion in NLP that continued pre-training LMs on task-related texts improves the performance of fine-tuning (FT) in downstream tasks. Through experiments on eight single-sentence tasks and eight sentence-pair tasks in both semi-supervised and fully-supervised settings, we find that conventional continued pre-training does not consistently provide benefits and can even be detrimental for sentence-pair tasks or when prompt-based FT is used. To tackle these issues, we propose Prompt-based Continued Pre-training (PCP), which combines the idea of instruction tuning with conventional continued pre-training. Our approach aims to improve the performance of prompt-based FT by presenting both task-related texts and prompt templates to LMs through unsupervised pre-training objectives before fine-tuning for the target task.

Organizers of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) today announced their Best Paper Awards, with the Best Overall Paper going to Beyond Accuracy: Behavioral Testing of NLP Models with CheckList by researchers from Microsoft, University of Washington, and University of California-Irvine. The winning paper introduces CheckList, a task-agnostic methodology for testing NLP models that includes a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, and a software tool that can quickly generate a large number of diverse test cases. ACL announced two Honourable Mentions in the Overall Best Paper category: Don't Stop Pretraining: Adapt Language Models to Domains and Tasks by researchers from the Allen Institute for Artificial Intelligence and University of Washington. The honourable mentions are Don't Stop Pretraining: Adapt Language Models to Domains and Tasks by researchers from the Allen Institute for Artificial Intelligence and University of Washington; and Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics by researchers from the University of Melbourne. The ACL 2020 Test-of-Time Awards meanwhile went to the 1995 papers Centering: A Framework for Modeling the Local Coherence of Discourse by Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein; and Unsupervised Word Sense Disambiguation Rivaling Supervised Methodsby David Yarowsky; and the 2010 papers Distributional Memory: A General Framework for Corpus-based Semantics by Marco Baroni and Alessandro Lenci; and Word Representations: A Simple and General Method for Semi-supervised Learning by Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio.